Adaptive color correction for pill recognition in digital images

ABSTRACT

A system and method for correcting colors in an image in order to identify pills. The system includes a surface that includes a background upon which pills are positioned and a border of known color adjacent to the background. The system also includes a digital camera and a processor. The processor is used to receive and process the image, use target objects on the surface to identify areas of the image that correspond to the known characteristics and features of the surface, perform image diagnostics to determine if the image can be used, and correct image color based on comparison of identified areas of the image that correspond with the known characteristics and features of the surface. Various graphical displays or cues can assist a user in obtaining a usable image.

CROSS REFERENCES

This application is a continuation of U.S. application Ser. No.13/665,720, filed Oct. 31, 2012, entitled, “SYSTEM AND METHOD OFADAPTIVE COLOR CORRECTION FOR PILL RECOGNITION IN DIGITAL IMAGES,” nowU.S. Pat. No. 8,948,508, which is incorporated by reference herein inits entirety for all purposes.

FIELD OF THE INVENTION

The disclosed embodiments relate to digital image processing foridentification of pills, and specifically to the correction of colors ina digital image so as to facilitate identification of pills.

BACKGROUND OF THE INVENTION

Pills of many shapes, sizes and colors are available as bothprescription and non-prescription medications. In the United States, thephysical identifiers of solid dosage pharmaceuticals are approved by theFederal Drug Administration. Ideally, no two pills are approved to haveexactly the same identifiers. Thus, pills are approved to each have aunique combination of shape, size, color, imprint (i.e., characters ornumbers printed on the medication), and/or scoring. Nevertheless,despite the fact that every type of FDA-approved pill is indeed intendedto be unique, the differences between pills is sometimes subtle. Forexample, two pills of the same shape but slightly different colorsand/or sizes may easily be confused by a patient. Pills normallydifferentiated by imprint may not appear to be different at all, forexample, if the imprints are not readable because the pills areface-down or the patient has poor vision. Such concerns are exacerbatedby the actions of patients who may not be fully coherent or alert.

Patients are not the only individuals who have a need to quickly andeasily identify pills. Relatives or caretakers of patients may also havesuch a need. Their need may stem from their responsibility to providethe correct pills to the patient, or simply from a desire to verify thatthe patient has taken the correct pills. Hospitals may have a need toquickly identify each of a collection of pills that a person brings fromhome or that may have been ingested by a child admitted for accidentalingestion of medication. Pharmacies have an interest in ensuring thatcorrect pills are dispensed. Insurance companies may even have aninterest in monitoring medication adherence, ensuring that correct pillsare dispensed to and taken regularly by the insured. In other words,many parties have an interest in verifying the identity of pills,whether the pills are identified individually or as a collection ofvarious pills.

Pills can be identified using various photographic and image processingmethods. For example, a digital image of a pill or collection of pillscan be taken, and then image processing methods can be used to determinehow many pills are in the image, the location and boundaries of thepills in the image, and to assign pixels in the image to a potentialpill for identification. This process of segmentation ideally results inevery pixel in the image either being assigned to a pill withwell-defined and accurate boundaries or being disregarded as notbelonging to any pill. Once pixels are assigned, the accumulated pixelsfor a given pill can be analyzed to determine the characteristics of thepill (e.g., its size, shape, color and imprint).

Color is an important feature used in object identification, especiallyin pill identification, where sometimes the color of the pills is one ofvery few differences between two different pills that have the same sizeand shape. However, the color of an object can appear different underdifferent lighting conditions. Thus, if the imaged color is notcorrected, misidentification of the pill can occur.

Recognizing this challenge, traditional pill identification techniqueshave sometimes required that ideal lighting conditions be used duringthe digital imaging process. Ideal lighting conditions can ensure thatthe imaged color is the actual color of the pills. Unfortunately,finding ideal lighting conditions that yield true color images aregenerally not convenient for most individuals. Additionally, traditionalcolor correction methods based on color distributions in a scene (suchas histogram equalization) generally do not result in good colorcorrection for pill identification.

Therefore, there is a need for a color correction method that cancompensate for less than ideal lighting conditions and automaticallycorrect imaged colors in pills prior to pill identification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a method of obtaining a color-correctable image forpill identification, in accordance with the present disclosure.

FIG. 2 illustrates a controlled surface for pill identification, inaccordance with the present disclosure.

FIG. 3 illustrates an image on a mobile device, in accordance with thepresent disclosure.

FIG. 4 illustrates method of performing image diagnostic tests, inaccordance with the present disclosure.

FIG. 5 illustrates an image being analyzed for shadows, in accordancewith the present disclosure.

FIG. 6 illustrates an image being analyzed for tilt, in accordance withthe present disclosure.

FIGS. 7A-7E illustrate visual cues used to obtain color-correctableimages, in accordance with the present disclosure.

FIG. 8 illustrates a mobile device system for correcting image colorswhile identifying pills, in accordance with the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof, and in which is shownby way of illustration specific embodiments that may be practiced. Itshould be understood that like reference numbers represent like elementsthroughout the drawings. Embodiments are described with sufficientdetail to enable those skilled in the art to practice them. It is to beunderstood that other embodiments may be employed, and that variousstructural, logical, and electrical changes may be made withoutdeparting from the spirit or scope of the invention.

A pill is a tablet, capsule, caplet or other solid unit of medication,prescription or over-the-counter, that is taken orally. Pills vary inappearance by color, size, and shape, among other features. Pillidentification through digital imaging and signal processing takesadvantage of these differences in pill appearances to identify a pill.For example, an individual can use a mobile device such as a smartphoneto image one or more pills. Software, resident either on the smartphoneand/or remote from the smartphone, processes the image to segment thepills, identify features of each imaged pill and then compare theidentified features of each pill with a database of pill features inorder to determine the identity of each pill. The pill database includesan indication of color for each pill in the database. The color of eachpill is standardized and is represented using one or more of severalcolor standards (for example, RGB, HSV, YUV, etc.). Thus, when one ormore pills are imaged, one or more color values for each pill aredetermined and then the determined color values are compared with thecolor values stored in the database. A match in color is one step inidentifying each pill.

Because varying lighting conditions can affect the values of anydetermined colors of imaged pills, and because mobile devices are notalways used in ideal lighting conditions, a method and system forautomatically correcting imaged colors is presented. The method issummarized in FIG. 1. In method 100, one or more pills are imaged on acontrolled surface with known characteristics and features (step 110).Once imaged, areas of the image are identified that correspond to theknown characteristics and features of the controlled surface (step 120).Image diagnostics are performed on the image in order to identify anyadverse imaging conditions (step 130). If identified imaging conditionsare within tolerances (step 140), the color of the image is correctedbased on comparison of identified areas of the image that correspondwith the known characteristics and features of the controlled surface(step 150). If identified imaging conditions are not within tolerances(step 140) and are therefore adverse, the user of the mobile device isnotified that a new image must be acquired. Various graphical displaysor cues can assist a user in obtaining an image within tolerances.

The basic premise of method 100 is to use parts or areas of an obtainedimage that correspond to parts of the controlled surface, where color isknown, to predict the colors of other parts of the image where thecolors are not known. In doing so, a first challenge is to accuratelydetect areas of an obtained image that correspond to known areas of thecontrolled surface. Because the obtained image is generally not obtainedunder ideal lighting conditions, the areas of the image that correspondto known areas of the controlled surface may still have a differentcolor (due to lighting conditions) and even a different shape (due tothe mobile device being at a non-standard angle or position). And yet,despite the differences in color and shape, the corresponding areas ofthe image must still be identified in order to determine any differencesand in order to apply the differences to the rest of the image.

Another challenge is to diagnose conditions that are adverse to theproper correction of color as well as to other pill detectionrequirements. In order to ensure short processing times, the colorcorrection model used in step 150 of method 100 typically includesassumptions and simplifications. This means, however, that certainadverse conditions may exceed the assumptions of the applied correctionmodel and thus render the resulting correction less accurate. Forexample, the color correction model may assume that the colors ofexternal lighting change smoothly across the imaged scene, thus removingthe need for a more complex predictive model. Thus, an imaged scene thatincludes an abrupt change in external lighting (a boundary betweenshadow and light, for example) would constitute an adverse conditionthat is not able to be handled by the selected correction model.Therefore, such adverse conditions must be identified and avoided. Otherexamples of adverse conditions include extreme lighting conditions,extreme camera angles, poor in-camera exposure, and poor camera focus.Real-time image diagnostics and visual cues are provided to determinewhether an image is even usable.

An additional challenge is to build a correction model that efficientlyand effectively corrects colors across an image, even if the image isobtained under complex (though not extreme) lighting conditions.

To solve these challenges, method 100 includes the use of a controlledsurface to aid in the identification of image areas that are used as abasis for the applied color correction. Various image diagnostics aretested to verify that any conditions are within tolerances. Real-timevisual cues are used to notify mobile device users of detecteddiagnostic problems. If the diagnostics testing completes without anyfinding of excessive adverse conditions, then an adaptive colorcorrection model is used that samples pixel values for color informationfrom specified image areas where corresponding values on the controlledsurface are known, calculates expected and actual color differences foreach pixel or set of pixels in the specified areas, and then constructsa statistical model of the differences. The model is used to predictcolor differences and to correct colors for one or more unknown areas inthe image.

The controlled surface 200 is illustrated in FIG. 2. The controlledsurface 200 includes a background 210 against which the pills areimaged. The background 210 is preferably a multi-color checkerboardbackground that includes a tightly-packed, closely-spaced matrix ofsquares. The high-density colored checkerboard pattern is used to createcontrast between an imaged pill and its background, which is essentialfor proper pill segmentation. The background 210 is surrounded by aborder 220 of known color characteristics. Typically, the border 220 isa known white. The border 220 may be other colors, but is preferablyachromatic. Because the color of border 220 is known, the correspondingborder areas of an image that includes the border 220 may be comparedwith the color characteristics of border 220 to determine anydifferences in color.

The controlled surface 200 also includes target objects 230, generallylocated in the border 220. The target objects 230 have specified colorand shape features in order to make them easily detectable. For example,in FIG. 2, the target objects 230 are four bulls-eyed yellow circleslocated near the corners of the background 210. The target objects 230include multiple high-contrast features. For example, in FIG. 2, thecenter of the bulls-eyed yellow circles is a black-filled circle with awhite dot in the middle. Other high-contrast patterns could be used inthe target objects 230. The high contrast features included in thetarget objects 230 are used for focus checking, as explained in greaterdetail below. Additionally, the high-contrast target objects 230 provideeasy-to-detect anchor points 240. The distances between these anchorpoints 240 on the surface are precisely known and can be used incontrast with distances in a corresponding image to calculate the scalefor the image.

Using controlled surface 200, corresponding surface features in an imagethat includes the controlled surface 200 can be detected and then usedfor object detection and recognition. To do this, a user of a mobiledevice such as a smartphone visualizes the controlled surface 200through a viewfinder or screen of a camera. In order to generate animage of appropriate scale, the anchor points 240, e.g., the white dotin the center of each target object 230, are required to be withincertain regions. This is accomplished by using software to identify theanchor points 240 in the center of the target objects 230. This is doneby first searching for the target objects 230 within certain targetregions. For example, when imaging the controlled surface 200, asillustrated in the smartphone- or camera-acquired image 300 of FIG. 3,the camera screen would display four target regions 330, one near eachcorner of the screen, in the locations of the image where the targetobjects 230 are most expected to be found. On-screen visual cues may begiven to users in real time to help the users align the camera orsmartphone so that each target object 230 is inside the correspondingtarget region 330. Because the target objects 230 are distinctive fromthe surrounding border 220 and background 210, and because of thehigh-contrast features included with each target object 230, standardcomputer vision techniques may be used by a camera to find the targetobjects 230 and determine whether the target objects 230 are withintarget regions 330.

Once the target objects 230 are located within the target regions 330,the anchor points 240 may be identified by locating the white dot at thecenter of each target object's black bulls-eye. Because the surfacepositions of the anchor points 240 on the controlled surface 200 areprecisely known, the known surface positions can be compared with theidentified anchor positions in the image 300, and then corrections canbe made to account for any geometric skew or distortions.

Once the anchor points 240 have been identified and located in the sceneto be imaged, various image diagnostics can be run in order to test foradverse conditions and ensure that any resulting image is acceptable forpill segmentation and identification. Thus, a number of differentdiagnostic tests may be run. Some tests are illustrated in method 400 ofFIG. 4. Method 400 represents a more detailed illustration of steps 130and 140 of method 100 of FIG. 1.

One diagnostic test in method 400 is an exposure check 410. Each targetobject 230 is checked to see if its predominate and known color (yellowin the example of FIG. 2) is within a predetermined range of colorvalues (depending on the color standard being used). The exposure testensures that color information is not lost due to extreme lightingconditions. In other words, it tests whether color can be restored orcorrected under the given lighting conditions. If the identifieddominant color in the target objects 230 is within the predeterminedcolor range, then the image's exposure is considered to be such thatcolors can be corrected.

Another diagnostic test in method 400 is a focus check 420. Each targetobject 230 is checked to determine if the percentage of high-contrastpixels within the target object exceeds a predetermined threshold. Ahigh-contrast pixel is a pixel that is at a boundary that exhibits achange in contrast. If an image is not in focus, the pixels will notrepresent a well-defined boundary and thus there will be a lowerpercentage of high-contrast pixels in each target object 230. However,if an image is in focus, a higher number of pixels in the target object230 will be high-contrast pixels. If a lower threshold is exceeded foreach target object, the image is considered to be focused.

An additional diagnostic test is a shadow detection test 430. One ormore strong shadows cast on the controlled surface 200 can make anyresulting image difficult to color correct. To determine whether strongshadows are present on the controlled surface 200, the achromatic border220 is sampled at periodic intervals around the border. FIG. 5, forexample, indicates possible sampling points 510 in image 500. The testthen determines any differences in light intensity for each of thesampled points 510 in the border 220. Differences in light intensity aredetermined between points in circular order, meaning in a sequentialorder around the border. If any differences in light intensity aresignificantly bigger than others, then the test assumes that a strongshadow is present and the image is not used. Thus, a predetermineddifference threshold is set, and if the difference between neighboringsampled points differs from any other differences between neighboringpoints by more than the predetermined difference threshold, then theimage is not used.

A fourth type of diagnostic test is a camera tilt angle detection test440. Camera tilt during image capture is determined by calculatingratios of distances between pairs of anchor points 240 in thegeometry-corrected image and comparing these ratios to predeterminedthresholds. For example, in the image 600 of controlled surface 200 ofFIG. 6, each anchor point 240 is a known distance from three otheranchor points. For example, distances 1, 2, and 3 are indicated asdistances between an anchor point and each of the other anchor points.An oppositely positioned anchor point is a distance 1′, 2′ and 3′ fromeach of the other anchor points. In an un-tilted image, the ratio ofdistances 1:1′, 2:2′ and 3:3′ would each be 1. When tilt exists,however, these ratios change. If any of the ratios are significantlydifferent (by a predetermined amount) from the un-tilted image ratios(of 1), then significant camera tilt is detected and the image is notused. While other methods may be used to determine tilt, such as using agyroscope, the above-described tilt angle detection test is advantageousbecause it does not require the mobile device to have a gyroscope orother tilt-detection technology. Additionally, the above-describedmethod works even when the controlled surface 200 is not itself level.

In order to assist a user of a mobile device attempting to acquire animage for pill identification, various real-time visual cues may bedisplayed to the user. The cues allow the user to acquire images thatwill pass each of the diagnostic tests applied to the images. Forexample, visual cues may be displayed on the mobile device's screen thatindicate when one or more target objects are out of focus. In FIG. 7A,for example, target objects that are out of focus (as determined by thefocus check 410) are outlined with a highlighted box (e.g, a red box)710, while target objects that are in focus are outlined with anon-highlighted box (e.g., green boxes) 720. As a user shifts the mobiledevice, the target objects can be moved into focus.

FIGS. 7B and 7C illustrate visual cues that indicate when the image istilted too much. Red bubbles 730 in the images of FIGS. 7B and 7Cindicate unacceptable camera tilt. In FIG. 7B, the unacceptable cameratilt is in the up/down direction as the image is oriented. Red bubble730 in FIG. 7B is like a bubble in a level and indicates that the camerais not level with regards to the controlled surface 200 in at least theup/down direction. In FIG. 7C, the unacceptable camera tilt is in theleft/right direction, according to the illustrated orientation of theimage.

FIG. 7D illustrates visual cues that may be shown when and where astrong shadow is detected. In the example of FIG. 7D, flashing small redsquares 750 indicate sampled points in the border 220 where the strongshadows are present (i.e., where the shadow detection test failed).

The presence of any of the visual cues communicates to the user that theimage is not acceptable. Thus, the user is motivated to adjust theimage, as cued by the display. Once adjusted, an acceptable image can beacquired, as illustrated in FIG. 7E.

Once an acceptable image is acquired, the image can be processed forcolor correction. This is done by sampling areas of the achromaticborder 220 which have known color values and comparing the sampledvalues to the known values. In order to improve the robustness of thesampling process and to avoid errors or noise resulting frominconsistencies on the border 220 (such as, for example, surfacespeckles, small scratches, or small dirt particles), samples of theborder 220 actually represent truncated averages of sets of pixels nearthe sample point. For example, if a sample is centered at point (x,y) onthe border 220, the resulting sample is assigned an average of the colorsample of each of the n-by-n pixels centered at the point (x,y). Incalculating the average color, outlying pixels are discarded. Thus, thesample for point (x,y) is a truncated average of n-by-n pixels. Multiplesample points are taken around the border 220.

Once a collection of sample points has been collected, the differencesbetween the sampled colors and the known color of the sampled pointsunder ideal lighting conditions is determined. By modeling the colorchanges across areas of known colors, a predictive mathematical modelcan be built to approximate the difference of the sampled color to thecolor of the sampled areas under ideal lighting for each pixel at point(x,y) as a function of its coordinates, as illustrated below in Equation1.D(L _(x,y) ,L _(x,y) ⁰)≈d _(x,y)=ƒ(x,y),(x,y)εS  Equations 1.

In Equation 1, L_(x,y) is the observed color/intensity of the pixel atcoordinates (x,y), L_(x,y) ⁰ is the color/intensity of the same pixelunder ideal lighting, D(,) is an intensity-color differential measurebetween the observed pixel and its appearance under ideal lighting(e.g., ratios of RGB channels), and S is the set of pixels in the border220 wherein the color of each pixel under ideal lighting conditions isknown. Function ƒ(x,y) is optimized such that the deviations d of allpixels in S are minimized according to some criteria (e.g., sum ofsquared errors). This can be done by standard statistical regressiontechniques. In an embodiment, function ƒ(x,y) is a polynomial.

Because the acquired image has passed any applied image diagnostictests, it can be assumed that the lighting across the entire image isrelatively smooth. Lighting across the image may not be uniform, but itmay still be smooth. Therefore, function ƒ(x,y), which assumes nodiscontinuities, can be used to predict color differences to be appliedto any pixel in the image. Of particular interest, function ƒ(x,y) canbe used to predict color changes to pixels belonging to pills onbackground 210 under ideal lighting conditions, as shown in Equation 2,below.D(L _(x,y) ,L _(x,y) ⁰)≈d _(x,y)=ƒ(x,y), for any (x,y) in theimage  Equation 2.

Because function ƒ(x,y) determines deviations d, pixel color can becorrected by summing the deviation d predicted by function ƒ at position(x,y) to the observed color of pixel (x,y). Once corrected, pills thatare identified through segmentation can be compared by color with thevalues of pill standardized colors, stored in, for example, a database.The database for comparison and pill identification may be located onthe smartphone or may be located remotely. In the case that thecomparison and pill identification database is located remotely, thesmartphone is configured to send data to the remote database forcomparison and identification.

Methods 100 and 400 are implemented as either hardware or software, or acombination thereof. The mobile device 800, as illustrated in FIG. 8,includes a system 850 for implementing methods 100 and 400. The system850 includes a color correction module to be used in conjunction withthe mobile device's camera, processor and a database. The mobile device800 generally comprises a central processing unit (CPU) 810, such as amicroprocessor, a digital signal processor, or other programmabledigital logic devices, which communicates with various input/output(I/O) devices 820 over a bus or other interconnect 890. The input/outputdevices 820 include a digital camera 822 for inputting digital images ofpills on the controlled surface 200. The input/output devices may alsoinclude a user interface 824 to display pill identification results to auser, and a transmitter 826 for transmission of the pill identificationresults to a remote location. A memory device 830 communicates with theCPU 810 over bus or other interconnect 890 typically through a memorycontroller. The memory device 830 may include RAM, a hard drive, a FLASHdrive or removable memory for example. The memory device 830 includesone or more databases. The CPU 810 implements the methods 100 and 400 asapplied to the digital image obtained by camera 822. The CPU 810processes the digital image, compares the colors identified for eachpill in the obtained image to one or more pill databases, at least oneof which is stored in the memory device 830, and then outputs pillidentification results via the user interface 824 and/or the transmitter826. If desired, the memory device 830 may be combined with theprocessor, for example CPU 810, as a single integrated circuit.

System 850 includes the color correction module 855. The colorcorrection module 855 performs methods 100 and 400. The color correctionmodule 855 can run various diagnostic modules such as an exposure checkmodule 860, a focus check module 865, a shadow detection test module 870and a camera tilt angle detection test module 875. In addition to thediagnostic tests identified in method 400, the color correction module855 can run additional tests 880 or provide additional visual cues to auser. As an example, system 850 and the modules used within system 850may be implemented as an application on a smartphone.

The above description and drawings are only to be consideredillustrative of specific embodiments, which achieve the features andadvantages described herein. Modifications and substitutions to specificprocess conditions can be made. Accordingly, the embodiments of theinvention are not considered as being limited by the foregoingdescription and drawings, but is only limited by the scope of theappended claims.

The invention claimed is:
 1. A method of correcting colors in an image,the method comprising: receiving a digital image representing one ormore pills on a surface, the digital image obtained using a digitalcamera and the surface; identifying areas of the image that correspondto areas of the surface with known colors and features; and correctingcolor in the image based on a comparison of image colors withcorresponding surface areas with known colors.
 2. The method of claim 1,wherein the surface includes a plurality of target objects which includeanchor points whose positions with respect to each other are known. 3.The method of claim 2, wherein identifying areas of the image thatcorrespond to areas of the surface with known colors and featurescomprises: identifying the anchor points in the image that correspond tothe anchor points on the surface.
 4. The method of claim 2, whereinidentifying areas of the image that correspond to areas of the surfacewith known colors and features comprises: comparing positions of anchorpoints in the image with positions of anchor points on the surface inorder to correct for geometric variations between the image and thesurface.
 5. The method of claim 1, further comprising: performing imagediagnostics on the image to identify imaging conditions adverse to colorcorrection.
 6. The method of claim 5, wherein the image diagnosticsinclude an exposure check.
 7. The method of claim 5, wherein the imagediagnostics include a focus check.
 8. The method of claim 5, wherein theimage diagnostics include a shadow detection test.
 9. The method ofclaim 5, wherein the image diagnostics include a camera tilt angledetection test.
 10. The method of claim 1, wherein correcting color inthe image comprises: using statistical regression applied to apredetermined function.
 11. The method of claim 1, further comprising:comparing the corrected colors of pills in the image with a database ofknown colors for pills.
 12. A system for correcting colors in an image,comprising: a surface for imaging one or more pills; and a processorconfigured to: receive a digital image representing the one or morepills on the surface; identify areas of the image that correspond toareas of the surface with known colors and features; and correct colorin the image based on a comparison of image colors with correspondingsurface areas with known colors.
 13. The system of claim 12, wherein thesurface comprises a plurality of target objects which include anchorpoints whose positions with respect to each other are known.
 14. Thesystem of claim 13, wherein the processor is further configured to:identify the anchor points in the image that correspond to the anchorpoints on the surface.
 15. The system of claim 13, wherein the processoris further configured to: compare positions of anchor points in theimage with positions of anchor points on the surface in order to correctfor geometric variations between the image and the surface.
 16. Thesystem of claim 12, wherein the processor is further configured to:perform an exposure check.
 17. The system of claim 12, wherein theprocessor is further configured to: perform a focus check.
 18. Thesystem of claim 12, wherein the processor is further configured to:perform a shadow detection test.
 19. The system of claim 12, wherein theprocessor is further configured to: perform a camera tilt angledetection test.
 20. The system of claim 12, wherein the processor isfurther configured to: compare the corrected colors of pills in theimage with a database of known colors for pills.